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ABSTRACT 

In this guide the logic of product evaluation is 
described in a framework that ia meent to be general and adaptable to 
all kinds of evaluations. Bvsluators should consider using the logic 
and methods of product evaluation when (1) the purpose of the 
evaluation is to aid •valuators in making a decision about purchases; 
(2) a comprehensive assessment of a product or program is important, 
and omitted information might be criticel; end (3) the evaluation is 
inherently comparative or suametive in nature. A product evaluation 
checklist is presented and discussed. Major considerations on the 
checklist include examining the needs of and resources available tor 
the consumer, looking at the range of possible products, studying the 
products in the light of needs and resources, and, through synthesis 
and summary, determining the best buy. Included in the discussion of 
each item on the checklist are specific questions, illustrative 
examples of how to apply the checklist, and warnings about ways to go 
wrong. (RM) 
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The World is too much with us 
Getting and spending we lay waste our 
powers. Wordsworth 



V4* 



Caveat Emptor 



In this guide the logic of product evaluation is described in 
a framework that is meant to be general and adaptable to all 
kinds of evaluations. 



KEY CHARACTERISTICS OF PRODUCT EVALUATION 



• Product evaluation aims to provide information that will aid 
decision-makers in getting maximum value for the money they 
spend* 

• Product evaluation is not done in a vacuum-- it considers the 
needs and resources of the consumer and the environment of 
use. 

m Product evaluation is inherently comparative in nature; a 

range of products rather than a single product is considered. 

• Product evaluation moves to an overall evaluation of worth or 
value and synthesizes independent ratings over many 
dimensions of performance. 



WHEN TO USE THE METHODS OF PRODUCT EVALUATION 

Kvaluators should consider using the logic and methods of 
product evaluation when 

• the purpose of the evaluation is to aid decision makers in 
making a buy/no-buy kind of decision; 

• a comprehensive assessment of a product or program is 
important and omitted information might be critical; 

• the evaluation is inherently comparative or summative in 
nature. 



AN EVALUATION CH EC KLIST 



The checklist shown on the following page is a comprehensive 
though idealized list of product evaluation points. Therefore, 
although few, if any, evaluations satisfy all the requirements of 
the checklist, it still has heuristic value. By addressing each 
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of the points included in the checklist, the ©valuator is 
unlikely to omit anything that would have critical bearing on the 
decisions to be made* 



The checklist described below is derived from, and is 
illustrated with, examples froa the field of product evaluation. 
It takes little effort to see that general concerns of the 
checklist apply equally well to program evaluation. Since 
product evaluation is a more refined and practiced art than 
program evaluation, it is worthwhile to study the practices in 
this field and extract what is applicable for other areas of 
evaluation. Each of the points in the checklist below is 
described more fully in the sections which follow. 



Product Evaluation Checklist 



Looking NEEDS 

st - what qualities are desirable? 

the RESOURCES 

Consumer - What can be drawn upon? 



Looking at the Range CRITICAL COMPETITORS 

of Possible Products - what's available? 



Studying the PERFORMANCE TESTS 

Products - How well do they work? 

in the Light FEATURES AND FLAWS 

of Needs - What else do they have/do? 

and Resources COSTS 

- Kow much do they really cost? 
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Bringing It SYNTHESIS AND SUMMARY 

All Together - Which is the best buy? 



DISCUSSION OF THE CHECKLIST 



Included in the discussion of each checklist point are 
defining questions, illustrative examples of how to apply the 
checklist points, and also warnings about ways to go wrong. 



NEEDS 



What qualities are desirable in this product* and to what 
degree? 



It is not enough to study only the products the evaluator 
must also look at the consumer and Identify his or her potential 
needs — a process about which the following notes are made. 



1. Needs are best determined by hypothesizing consumer 

characteristics and carrying out an analysis of desired 
performance functions. Sometimes an idealized product can be 
specified in a functional way, such as this description from 
Consumer Reports ; 

a good barbecue sauce should not be so strongly 
flavored as to obliterate the flavor of the meat 
itself, nor should it be so mild as to add 
nothing . . . 

Sometimes clarification and definition of product functions are 
required t 

antiseptic - used for washing around the wound 
wound cleanser - for cleaning the wound itself 
ointments - protecting the wound from further 
infection 

Consumer Reports, July 1984 



2. When possible, it is most useful to specify needs in terms of 
quantifiable dimensions: 

a timekeeping device is needed which is small and 
portable (wristwatch) , which is « t, irate to 15 
seconds per day, and waterproof (r. .j 100 feet). 

Consumer Reports, July 1984 



3. Evaluation is most straightforward when needs are stated 
directly in performance terms. Usually, however, needs are 
not '.ell understood nor can they be clearly stated. The 
evaluation must then begin with an initial attempt to 
determine, and state at least loosely, the priority needs. 
Then, a cyclical process of testing (learning about the 
product}, restating the need, more testing, restating the 
need . . . can follow. 

Example s 

1. Stating the need s A boat is needed that is fit enough 
to sail the oceans of the world. 

2. Learning about the product ? Expert sailors are 
consulted as to what makes a well-founded boat. 



3* Restating the needs A boat Is needed that baa (a) the 
ability to right itself in a knockdown, (b) the ability 
to claw off a lee shore in a storm, (c) features of 
self-steering and easy handling, and (d) an average 
cruising speed of 5 or more knots. 



Ways to Go wrong 

1. Overly rigid definition of the users 

Consumer Reports (CR) gave the Minolta 110 camera a negative 
evaluation as it was significantly bulkier and heavier than 
other 110 cameras. The Minolta, however, did possess zoom 
capabilities and performed as well as many of the larger 35mm 
zoom cameras. If Consumer Reports had included the larger 
room audience (both 3 Sow and 110 cameras), the Minolta would 
have received high ratings. 

2. Inaccurate conception of uses 

Consumer Reports rated a Norelco midget recorder behind other 
models because it had poor long-range pickup of music or 
talks. In this evaluation, CR missed the point that the 
recorder was designed primarily for dictation where 
elimination of background noise is essential. Thus, it was 
by design that it failed on tests of long-range pickup. A 
brief survey of users or a functional analysis would have 
revealed the mistake. 



RESOURCES 

• What resources are available to obtain, maintain* or 
improve the chosen product ? 

A second look at the consumer is required here. The 
evaluator must look at the product not only in terms of what 
consumers need, but also in terras of what resources they may 
already possess. 

1. Money: 

Cash or credit is usually needed for a purchase. It is 
useless to determine that a $25,000 Mercedes is the best buy 
when consumers have only $7,000 in their budget. Many 
evaluations will divide the products to be surveyed into cost 
categories before beginning comparisons. 

2. Other products: 

There say already be, on the shelves or in service, a product 
that can fill the needs of the consumer. This is especially 



true when the needs are stated loosely in terns of desired 
functions, rather than in terras of a specific brand or model. 

Examples 

In order to get a small stereo system, a consumer may 
wish to purchase adapters and speakers to supplement 
an already-owned Sony Walkman. This- may be an 
equally effective and much cheaper solution than 
buying an entire small stereo system. 

3. Skills: 

Especially when considering high technology products, the 
presence of skilled operators and maintenance people may 
significantly affect the choice of purchase. Also, the 
feasibility of adapting an existing product may depend upon 
the availability of skilled people to do the work. 

4. Environmental supports and cons* raints: 

The existence of other interacting systems or equipment, or 
potentially compatible systems, may alter the choice of 
product to be purchased. 

Example: 

The lack of 220 volt power may greatly limit the 
choices in deciding which clothes dryer to purchase. 



CRITICAL COMPETITORS 

o Which products are available in the range defined by 
the relevant needs and resources ? 

Product evaluation is by nature comparative and judgmental. 
It is not enough for an evaluator, for example, to decide that an 
IBM Selectnc III is the best buy because it meets the needs of 
the consumer and falls within the limits of the resources 
available. To evaluate only one machine may neglect a far 
superior buy in terms of quality or price, or both. Comparison 
is required. 



A "critical competitor" is any product that does as well in 
terms of its outcome relative to its costs, and relative to the 
needs and resources of the consumer. Critical competitors are 
alternate solutions to the problem posed by the consumers* 
unfilled needs and limited resources. Critical competitors come 
in the following four types. 
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1. The obvious competitors* 

These include market competitors and nearby relatives. For 
example, the competitor of the Volkswagen (VW) would be other 
compacts of similar price, features, and per for nance 
characteristics. Also, one sight want to consider 
comparisons with other current VW models, or even earlier 
models. 

2. The eaergent competitors i 

As one tests a product, the discovery of significant 
side-effects or unexpected functions say lead to the 
reconceptualization of the product itself. This, in turn, 
say call for new competitors and comparisons. 

Examples i 

A new algebra textbook is so well written and full of 
examples that it appears to lead to significant 
reading gains. Comparison with other more standard 
readers is called for. 

A photographic copier produces copies of such high 
quality that it is competitive with commercial 
printers — with which it should now be compared. 

3. The radical critical competitor: 

By going back to basic needs (the desired performance 
functions stated in most general terras) , an entirely 
different notion of competitor might be conceived. 

Examples 

Poor design in a newly completed building resulted in 
having insuff icie *.t elevators to serve rush hour 
crowds. The problem was exacerbated by the lack of 
indicators in the lobby, which resulted in greater 
uncertainty and irritation among the waiting would-be 
elevator patrons- The cost of installing a new 
elevator (or even indicators) in a completed building 
was exorbitant. Clearly, an alternate solution, a 
critical competitor for an elevator, is called for. 

By reeonceptualising the situation, we can see that 
the problem was twofold i (1) people needed to be 
moved faster, and (2) the irritation had to be 
remedied. Addressing the second of the two aspects 
of the problem, full-length mirrors were installed on 
both sides of the elevator, and on the wall facing 
the doors. In a testament to the human ego, the 
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level of irritation dropped radically. Thus, a 
mirror turns out to be a clever and inexpensive 
critical competitor for an elevator indicator . . . 

4. The invented competitors 

At times the evaluator may wish to create — think up and 
develop— a suitable competitor. In the absence of a suitable 
horse to run against* the evaluator may have to 90 to some 
lengths to provide an optimal competitor. 

Examples 

In testing computer assisted instruction (CAD* an 
evaluator could not find a similar CAI program to 
compare with. By removing the instruction from the 
computer and putting it in the form of a programmed 
text, the evaluator created an excellent competitor. 
Not only was it much cheaper , but it gave a way of 
quite directly assessing the contribution of the 
machine to the learning process. This can be an 
embarrassing test for any computer based learning 
program, as .it often produces equal learning gains at 
a fraction of the cost. 



The need to test (sometimes even to invent and develop) 
critical competitors increases the work and expense of the 
evaluation. Judicious and parsimonious decisions need to be made 
in the selection of competitors. Zt should also be noted that in 
using a comparative approach, one may or may not discover why one 
product is better than another, but this is rarely crucial for 
the consumer to know. It is simply easier, at times, to run 
comparison tests than it is to make absolute measurements of 
performance differences. (For example, 12 meter yachts are tuned 
for the America's Cup by constant comparison with a second trial 
yacht which is held constant. Small increments of speed in 
varying conditions are simply too hard to measure in absolute 
terms. ) 



ways to Go Wrong 

1. No competitors 

Like doing an experiment without a control group, the lack of 
a competitor leaves many questions unanswered. 

2. Accepting the market definition of the competitor: 
Examples 

; when commercially available rug cleaners were tested 

against each other, little difference in price or 
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performance) was noted. In a moment of skepticism, an 
•valuator compared then against a weak solution of 
Tide detergent. Tide was found to be more effective 
at a fraction of the cost. 

3. Restricting the rcnge of competitors t 

The earlier example of the low rating of the Minolta 110 
camera can be seen aa a failure to consider the right 
competitors. When compared with 110 cameras, the Minolta 
appeared to be excessively bulky. When 35mm cameras were 
included, the cooper ison was far more favorable. The close 
relationship between the choice of critical competitors and 
the definition of consumer needs is obvious. 



PERFORMANCE TESTS 

c Which products have which performance qualities, and 
to what degree ? 

How well does it work? Whet functions, as specified by the 
needs analysis, are met by the product (s) and how well? 



Zn evaluating performance the evaluate* must decide which 
variables to test, to what degree of precision they should be 
tested, and what assumptions can be made about the potential 
users. Also, whenever possible, products should be tested in 
different modes of operation—in a way that measures maximum 
performance, in a way that the product is likely to be used, and 
finally, if different, in a way that corresponds to the 
manufacturer's use directions. 



Sometimes the ultimate desired performance of a product can 
be broken down into specific component capabilities. 



In evaluating FM receiver capabilities, CR rates the 
performance of each receiver in terms of its 
sensitivity (ability to pick up weak signals) , its 
capture ratio (the ability to reject weak interfering 
signals) , the frequency response (bow different 
frequencies are received) , freedom from distortion 
and stereo separation. (See Table 1.) 



Example s 




SEST COPY AVAILABLE 
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Table 1 

Breaking Down Performance lata Component Qualities 
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Mid-priced stereo receivers 



Ustsd in order of estimated overall 
quality; whsre overall scores are 
equal, listed alphabetically. Differ- 
ences in score of 7 points or less were 



Judged insignificant. Ratings should 
be read m conjunction with table of 
specifications and features on pages 
380-381. 
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ttaya to Go Wrong 



1. The technist fallacy: 

Sometimes products are tested for differences in levels of 
performance that far exceed any differences ever likely to be 
noticed in actual use. For example, stereo Review ranks 
stereos according to differences in measures of fidelity that 
cannot be detected by the human ear. In the technist 
fallacy, distinctions are made where differences do not exist. 

2. Features over functions: 

Under the guise of performance testing, features may be 
described while performance functions are ignored. Sailing 
journals are particularly guilty of this error in their 
"performance tests" of new production boats. Their 
evaluation articles contain elaborate descriptions of deck 
and interior features, some discussion of design, little 
information about performance, and usually no test data about 
seaworthiness in difficult seas. 

3. Poor sampling: 

Sometimes nonrepresentative products are tested, or 
alternatively, products are tested without sufficient 
sampling, if the variance among the cars of a particular 
make is greater than the variance between makes, and the 
performance tests use only one car of each make, then such 
tests will give the consumer almost no reliable information. 

4. Restricting the range of consumers: 

The potential users must be considered in designing 
performance tests. When testing stereo equipment for high 
priced-cars, the tests failed to take into consideration that 
the purchasers of such equipment are likely to be older 
people who have significantly higher frequency hear in? loss. 
If the testers, as is likely, are younger, the true consumer 
may be misrepresented. 

5. Restricted range of environments: 

in Northern Vermont, the way a car performs on icy and 
snow-covered roads is of great concern to consumers, and is 
thus a key variable in their purchasing decisions. Few car 
evaluations provide such climate-related data. 

When the Environmental Protection Agency (SPA) tests cars for 
their comparative fuel consumption (apg ratings) , they 
conduct the tests in the lab on a dynamometer. Such a 
machine cannot take into account wind resistance, and thus 
the SPA measures are, at best, very rough estimates. They 
are often at wide variance with the results of other more 
realistic toad tests. 

» 12 



FEATURES AND FLAWS 

o What secondary features, functions, or 
character 1st tea do the products possess? 

' In addition to basic performance characteristics, products 
are often compared as to their special features and flaws. 
Advantages and disadvantages in terms of convenience , comfort, 
feel, style, complexity, sturdiness, noise, smell, etc., can be 
compared. 

Examples i 

In an evaluation of dishwashing liquids, a dermatoloo 1st 
was asked to rate the therapeutic or irritating effect 
of the liquids tested on the hands. 

The rear wiper of the Volvo is given high marks as a 
sensible and very convenient feature. 

The plastic-coated handles of Sears' slip- joint pliera 
are recommended as a comfort feature. 



In discriminating between products, often the detailing of 
the flaws discovered in a product is even more helpful than the 
listing of features. The following statements are from Consumer 
Reports (July 1984). 

Examples: 

The climate control system required some fiddling, and 
the radio was a nightmare of needless complexity. 

The milkshake was very thick and foamy, leaving an 
unpleasant coating in the mouth; it tasted strongly of 
artificial chocolate and was quite sweet. 

The Lincoln accumulated 11 sample def acts. The 
drivel ine vibrated at expressway speeds. The right door 
lock was defective, and there were many cosmetic flaws 
inside and outside the car. We expected more from a car 
in this price range. 



The consideration of flaws in a product can be very important 
and can even override all other considerations when they make the 
product unsafe or otherwise unacceptable s 

Examples: 

In its evaluation of antiseptics, CR points out that the 
long-used favorite, Hercurochrome, was found to be 
unsafe, or ineffective, or both. 
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CR wiii, from time to time, rate cars as unacceptable 
because of poor handling under emergency conditions, or 
will rate an electrical appliance as unacceptable 
because of potential shock hazard. 

A textbook say be judged unacceptable if it is found to 
be racist or sexist. 



COSTS 

e What are the true costs of the products? 

It is not enough to report the list price of the product as 
the cost of the product. Since the cost of th* product is half 
oC the input in determining a best-buy , the consumer needs to 
have as complete and accurate a picture of costs as possible. 
The following notes reflect some refinements to determining 
actual costs. 

1. The true price of the products 

Very often substantial discounts are available. While it is 
not usually possible to state specific prices, the evaluator 
may be able to indicate the range of discounts available. 
Also, many list prices do not include delivery charges, 
dealer-prep charges, taxes, options, or other mark-ups. 

2. Maintenance costs: 

Estimates can be made about repair frequency and repair 
costs. For some products the costs of parts and labor may 
approach the original purchase costs in a very short time. 
(For example, safety-blade razors are often sold at a very 
low price as their purchase forces the consumer to buy that 
brand of replacement blades which are sold at considerable 
profit.) 

3. Support costs s 

Operating costs, energy costs, or other support costs can be 
estimated. 

4. Additional costs or cost savings can be noted: 

- Warranties can vary greatly in their coverage and time 
limitations. 

- Certain products may qualify for special tax treatment. 

- Certain products are likely to have particularly high (or 
low) insurance costs. 

- Some products have traditionally strong resale value. 




Ways to Go wrong 



1. Comparing nonequivalent versions: 

A 10-year-old used cruising boat may cost $50,000. For the 
same price one may be able to buy a new boat "ready to 90. * 
However , when the production boat is actually brought up to 
the standards of the used boat in terms of safety, special 
features , and backup equipment, the cost of the new boat may 
have nearly doubled. 

2. Comparing nonequivalent quantities: 

Sixteen ounces of the best dishwashing liquid will wash five 
times as many dishes as the least effective dishwashing 
liquid. To compare the cos t-per -ounce of these two products 
would be highly misleading. 

3. Overestimating salient cost features: 

Consumers often use mpg ratings to infer the relative costs 
of operating automobiles. Zf one drives 15,000 miles per 
year, the difference between a 35 mpg and 45 mpg car is only 
about $120 per year— a cost difference that may be much lower 
Lnan other less obvious cost factors. 



SYNTHESIS AND SUMMARY 

• Which product is the best buy? 

How do we pull it all together to come to an overall judgment? 

Consumer Reports ( CR ) uses several different approaches to 
summarizing and making recommendations. In some evaluations CR 
lists products in order of overall quality, with comparative 
component ratings shown on different performance characteristics 
and features. (See Table 1.) CR also separates out 
nonacceptable products because of very low quality or a safety 
hazard. In some evaluations, CR will divide products into two 
levels of quality, giving a "check rating" to those items that 
are overall high quality and judged to be superior to none heck 
rated items. CR also gives * "best buy" rating to those products 
judged to be "of overall high quality and of relatively low 
price." Exactly bow these judgments are made is not detailed, 
but CR claims it is a mixture of controlled testing and expert 
judgment. 
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The weighted Sum 



The classic way to combine component ratings to arrive at ar. 
overall rating ia to use a weighted-sum approach. This means 
that each performance and feature distension is given a weight 
proportional to its overall importance f the rating in each 
dimension is then multiplied by this weight and the products are 
then summed to give an overall rating for each item tested. 



There are several difficulties with this approach: 

1. It is not obvious *ow to decioe the weighting scale for the 
performance and feature dimensions , nor is it easy to decide 
what weightings should be given. 

2. There is rarely a linear relationship between the util tv 
(the amount a need is satisfied and the importance of that 
need) and the performance measure. Thus, one often has the 
uneasy feeling that somehow this mathematical summation is 
too simplistic is missing something important but 
ineffable. 



In spite of these difficulties, the weighted-sum approach can 
be useful, and tiw following suggestions are made for its 
effective uses 

1. Simplify at the >eginning by deciding which performances are 
absolutely ease tial. Some coppeH ^ors may be eliminated 
bee use they fa 1 to provide some critical function, or they 
fall below a ve tain standard. 

2. 3 cart by assigning weights in a ratio way. "This is twice as 
important as that; these two are the same importance," and so 
on. 

3. With the weightings assigned, carry out the comparison of all 
the products. Bow do the final rankings agree with your 
intuition about them? If they are different, note what 
weightings cause the difference — perhaps they should be 
changed. 

4. Weightings can now be changed. Perhaps one dimension is as 
important a- i whole class of other dimensions or as all 
other dimensions together. Redo the ratings and again 
compare with your intuition about them. 

The point here is that the weighted sum approach and one's 
intuition complement each other. By using one as a check against 
the other, both become clearer. By going back and forth they 
should converge until they blend into each other. 



16 

is 



Other Methods 



There ere, of course, other ways of arriving at summary 
comparisons, and there is much to be said for giving the consumer 
many different ways of seeing the comparisons: 

1. The match play approach: 

Final candidates are compared against each other in 
head-to-head competition. By elimination or "round-robin" 
comparisons, one gains a different perspective on the 
relative strengths and weaknesses of the products. 

2. Expert judgment: 

Those well-versed in trie area of the products being studied 
can b" solicited for their judgments as to overall quality 
based on their own experience, and, or they can review the 
weightings and procedures use*, in a weighted-sura approach. 

3. "aecdotal evidence: 

The description of an expert race-car driver comparing the 
performances ot cars on an obstacle course may be of great 
value to sofae consumers. This kind of connoisseur's 
appraisal may be translated into numerical rating, or perhaps 
more instructively kept in narrative form. 

4. The naive consumer: 

Like the expert, naive consumers have a unique and valuable 
perspective to offer. They can be used to review a 
weighted-sum approach, or to give their own narrative 
evaluations. 



The evaiuatoir does much for the consumer when he/she does the 
work of synthesizing the component ratings. The use of multiple 
methods and an iterative converging procedure minimizes the 
chance of being far off the mark. If the overall product ratings 
are supplemented by component ratings, pair-wise comparisons, and 
expert and novice testimony, then the consumer has a strong basis 
for making an intelligent purchasing decision. 



scriven, N. (1981). Product evaluation. In H. L. Smith (Ed.), 
Hew techniques for evaluation. Beverly Hills, CA: Sage 
Publications. 
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